Environment Object Recognition

ABSTRACT

Methods, systems, and devices for outputting additional content to a user are disclosed. One or more image capture devices may be controlled to capture images of areas where users might be viewing content. Objects in the captured images may be recognized. Based on the results of object recognition, additional content may be determined and output for display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to co-pendingU.S. patent application Ser. No. 13/775,956, which was filed Feb. 25,2013 and entitled “ENVIRONMENTAL OBJECT RECOGNITION” and which is herebyincorporated by reference herein in its entirety.

BACKGROUND

Content provided to consumers often comes in a one-size-fits-allvariety. That is, content is regularly transmitted to the consumingpublic as a whole, not tailored to the end consumer. Many consumers whoultimately receive the content may not relate to or enjoy what is beingoffered. Content providers, advertisers, and the like, have conductedresearch to determine, e.g., an average demographic group that may beviewing content at a particular time or location and then tailoredcontent based on the average viewing public. Such an approach, however,alienates a significant portion of consumers who are accessing contentbut who do not fall into the average demographic. There remains a needto gather information about a consumer, and tailor content provided tothat consumer accordingly. This disclosure relates to providing contentto a user in a way that overcomes the above shortfalls.

SUMMARY

The following summary is for illustrative purposes only, and is notintended to limit or constrain the detailed description.

The disclosure is directed to methods, systems, and devices aimed atsatisfying the need of delivering targeted content to end users. Someaspects of this disclosure relate to capturing images of objectsassociated with a user accessing content, and providing the useradditional content based on the captured images. Further aspects of thisdisclosure relate to tracking a user's interaction with content, such asby, e.g., determining which portion of delivered content a user isviewing, and providing additional content based on the user'sinteraction.

In an exemplary embodiment of the present disclosure, this is achievedby associating an image capture device (e.g., a camera) with a displaydevice. The image capture device captures an image of a user accessingcontent and her surroundings, and at least one object in the capturedimage is recognized. In another aspect, what is being viewed by theuser's eyes may be determined, and an object on the display screen whichthe user is viewing is recognized. The recognized object may be comparedto a list of known or dynamically determined target objects, and, if itis determined the object matches a target object, additional content maybe selected based on the match. The additional content may then bepresented or offered to the user. In this way, targeted content isprovided to the user according to objects she is near or interactingwith while accessing content, further tailoring the content to theuser's individual tastes.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the presentdisclosure will become better understood with regard to the followingdescription, claims, and drawings. The present disclosure is illustratedby way of example, and not limited by, the accompanying figures in whichlike numerals indicate similar elements.

FIG. 1 illustrates an example communication network on which variousfeatures described herein may be used.

FIG. 2 illustrates an example computing device and softwareconfiguration that can be used to implement any of the methods, servers,entities, and computing devices described herein.

FIG. 3 illustrates a user's room as an example environment in which someembodiments of the present disclosure may be used.

FIG. 4 illustrates an example captured image of the room depicted inFIG. 3.

FIG. 5 illustrates recognition of example objects in the captured imageof FIG. 4.

FIG. 6a illustrates one embodiment of content displayed on a displaydevice.

FIG. 6b illustrates one embodiment of additional content associated witha recognized object from FIG. 5.

FIG. 6c illustrates another embodiment of additional content associatedwith a recognized object from FIG. 5.

FIG. 7 illustrates one embodiment of additional content associated withmore than one recognized objects from FIG. 5.

FIG. 8 illustrates another embodiment of additional content associatedwith more than one recognized objects from FIG. 5.

FIG. 9a illustrates one embodiment of content comprising a object.

FIG. 9b illustrates one embodiment of additional content associated witha object.

FIG. 10 illustrates recognition of example objects in an image capturedlater in time than the captured image of FIG. 4.

FIG. 11 illustrates one embodiment of additional content associated withmore than one recognized objects over time.

FIG. 12 illustrates one embodiment of displaying additional contentassociated with a plurality of recognized objects from a plurality ofusers.

FIG. 13 illustrates another embodiment of displaying additional contentassociated with a plurality of recognized objects from a plurality ofusers.

FIG. 14 illustrates a user's room comprising more than one displaydevice as an example environment in which one embodiment of the presentdisclosure may be used.

FIG. 15 illustrates one embodiment of displaying additional content onmore than one display device associated with a user.

FIG. 16a illustrates one embodiment of an example user interfaceassociated with one embodiment of the disclosure.

FIG. 16b illustrates another embodiment of an example user interfaceassociated with one embodiment of the disclosure.

FIG. 16c illustrates yet another embodiment of an example user interfaceassociated with one embodiment of the disclosure.

FIG. 16d illustrates yet another embodiment of an example user interfaceassociated with one embodiment of the disclosure.

FIG. 17 illustrates an exemplary flowchart of a method in accordancewith one embodiment of the disclosure.

DETAILED DESCRIPTION

In the following description of various illustrative embodiments,reference is made to the accompanying drawings, which form a parthereof, and in which is shown, by way of illustration, variousembodiments in which aspects of the disclosure may be practiced. It isto be understood that other embodiments may be utilized, and structuraland functional modifications may be made, without departing from thescope of the present disclosure.

FIG. 1 illustrates an example communication network 100 on which many ofthe various features described herein may be implemented. Network 100may be any type of information distribution network, such as satellite,telephone, cellular, wireless, etc. One example may be an optical fibernetwork, a coaxial cable network, or a hybrid fiber/coax distributionnetwork. Such networks 100 use a series of interconnected communicationlinks 101 (e.g., coaxial cables, optical fibers, wireless, etc.) toconnect multiple premises 102 (e.g., businesses, homes, consumerdwellings, etc.) to a local office or headend 103. The local office 103may transmit downstream information signals onto the links 101, and eachpremises 102 may have a receiver used to receive and process thosesignals.

There may be one link 101 originating from the local office 103, and itmay be split a number of times to distribute the signal to variouspremises 102 in the vicinity (which may be many miles) of the localoffice 103. The links 101 may include components not illustrated, suchas splitters, filters, amplifiers, etc. to help convey the signalclearly. Portions of the links 101 may also be implemented withfiber-optic cable, while other portions may be implemented with coaxialcable, other lines, or wireless communication paths.

The local office 103 may include an interface, such as a terminationsystem (TS) 104. More specifically, the interface 104 may be a cablemodem termination system (CMTS), which may be a computing deviceconfigured to manage communications between devices on the network oflinks 101 and backend devices such as servers 105-107 (to be discussedfurther below). The interface 104 may be as specified in a standard,such as the Data Over Cable Service Interface Specification (DOCSIS)standard, published by Cable Television Laboratories, Inc. (a.k.a.CableLabs), or it may be a similar or modified device instead. Theinterface 104 may be configured to place data on one or more downstreamfrequencies to be received by modems at the various premises 102, and toreceive upstream communications from those modems on one or moreupstream frequencies.

The local office 103 may also include one or more network interfaces108, which can permit the local office 103 to communicate with variousother external networks 109. These networks 109 may include, forexample, networks of Internet devices, telephone networks, cellulartelephone networks, fiber optic networks, local wireless networks (e.g.,WiMAX), satellite networks, and any other desired network, and thenetwork interface 108 may include the corresponding circuitry needed tocommunicate on the external networks 109, and to other devices on thenetwork such as a cellular telephone network and its corresponding cellphones.

As noted above, the local office 103 may include a variety of servers105-107 that may be configured to perform various functions. Forexample, the local office 103 may include a push notification server105. The push notification server 105 may generate push notifications todeliver data and/or commands to the various premises 102 in the network(or more specifically, to the devices in the premises 102 that areconfigured to detect such notifications). The local office 103 may alsoinclude a content server 106. The content server 106 may be one or morecomputing devices that are configured to provide content to users attheir premises. This content may be, for example, video on demandmovies, television programs, songs, text listings, etc. The contentserver 106 may include software to validate user identities andentitlements, to locate and retrieve requested content, to encrypt thecontent, and to initiate delivery (e.g., streaming) of the content tothe requesting user(s) and/or device(s).

The local office 103 may also include one or more application servers107. An application server 107 may be a computing device configured tooffer any desired service, and may run various languages and operatingsystems (e.g., servlets and JSP pages running on Tomcat/MySQL, OSX, BSD,Ubuntu, Redhat, HTML5, JavaScript, AJAX and COMET). For example, anapplication server may be responsible for collecting television programlistings information and generating a data download for electronicprogram guide listings. Another application server may be responsiblefor monitoring user viewing habits and collecting that information foruse in selecting advertisements. Yet another application server may beresponsible for formatting and inserting advertisements in a videostream being transmitted to the premises 102. Although shown separately,one of ordinary skill in the art will appreciate that the push server105, content server 106, and application server 107 may be combined.Further, here the push server 105, content server 106, and applicationserver 107 are shown generally, and it will be understood that they mayeach contain memory storing computer executable instructions to cause aprocessor to perform steps described herein and/or memory for storingdata.

An example premises 102 a, such as a home, may include an interface 120.The interface 120 can include any communication circuitry needed toallow a device to communicate on one or more links 101 with otherdevices in the network. For example, the interface 120 may include amodem 110, which may include transmitters and receivers used tocommunicate on the links 101 and with the local office 103. The modem110 may be, for example, a coaxial cable modem (for coaxial cable lines101), a fiber interface node (for fiber optic lines 101), twisted-pairtelephone modem, cellular telephone transceiver, satellite transceiver,local wi-fi router or access point, or any other desired modem device.Also, although only one modem is shown in FIG. 1, a plurality of modemsoperating in parallel may be implemented within the interface 120.Further, the interface 120 may include a gateway interface device 111.The modem 110 may be connected to, or be a part of, the gatewayinterface device 111. The gateway interface device 111 may be acomputing device that communicates with the modem(s) 110 to allow one ormore other devices in the premises 102 a, to communicate with the localoffice 103 and other devices beyond the local office 103. The gateway111 may be a set-top box (STB), digital video recorder (DVR), computerserver, or any other desired computing device. The gateway 111 may alsoinclude (not shown) local network interfaces to provide communicationsignals to requesting entities/devices in the premises 102 a, such asdisplay devices 112 (e.g., televisions), additional STBs 112, personalcomputers 114, laptop computers 115, wireless devices 116 (e.g.,wireless routers, wireless laptops, notebooks, tablets and netbooks,cordless phones (e.g., Digital Enhanced Cordless Telephone—DECT phones),mobile phones, mobile televisions, personal digital assistants (PDA),etc.), landline phones 117 (e.g. Voice over Internet Protocol—VoIPphones), and any other desired devices. Examples of the local networkinterfaces include Multimedia Over Coax Alliance (MoCA) interfaces,Ethernet interfaces, universal serial bus (USB) interfaces, wirelessinterfaces (e.g., IEEE 802.11, IEEE 802.15), analog twisted pairinterfaces, Bluetooth interfaces, and others.

FIG. 2 illustrates general hardware elements that can be used toimplement any of the various computing devices discussed herein. Thecomputing device 200 may include one or more processors 201, which mayexecute instructions of a computer program to perform any of thefeatures described herein. The instructions may be stored in any type ofcomputer-readable medium or memory, to configure the operation of theprocessor 201. For example, instructions may be stored in a read-onlymemory (ROM) 202, random access memory (RAM) 203, removable media 204,such as a Universal Serial Bus (USB) drive, compact disk (CD) or digitalversatile disk (DVD), floppy disk drive, or any other desired storagemedium. Instructions may also be stored in an attached (or internal)hard drive 205. The computing device 200 may include one or more outputdevices, such as a display 206 (e.g., an external television), and mayinclude one or more output device controllers 207, such as a videoprocessor. There may also be one or more user input devices 208, such asa remote control, keyboard, mouse, touch screen, microphone, etc. Thecomputing device 200 may also include one or more network interfaces,such as a network input/output (I/O) circuit 209 (e.g., a network card)to communicate with an external network 210. The network input/outputcircuit 209 may be a wired interface, wireless interface, or acombination of the two. In some embodiments, the network input/outputcircuit 209 may include a modem (e.g., a cable modem), and the externalnetwork 210 may include the communication links 101 discussed above, theexternal network 109, an in-home network, a provider's wireless,coaxial, fiber, or hybrid fiber/coaxial distribution system (e.g., aDOCSIS network), or any other desired network.

FIG. 2 is an example computing device having a hardware and/or softwareconfiguration. Modifications may be made to add, remove, combine,divide, etc. components of the computing device 200 as desired.Additionally, the components illustrated may be implemented using basiccomputing devices and components, and the same components (e.g.,processor 201, ROM storage 202, display 206, etc.) may be used toimplement any of the other computing devices and components describedherein. For example, the various components herein may be implementedusing computing devices having components such as a processor executingcomputer-executable instructions stored on a computer-readable medium,as illustrated in FIG. 2. Some or all of the entities described hereinmay be software based, and may co-exist in a common physical platform(e.g., a requesting entity can be a separate software process andprogram from a dependent entity, both of which may be executed assoftware on a common computing device).

One or more aspects of the disclosure may be embodied in acomputer-usable data and/or computer-executable instructions, such as inone or more program modules, executed by one or more computers or otherdevices. Generally, program modules include routines, programs, objects,components, data structures, etc. that perform particular tasks orimplement particular abstract data types when executed by a processor ina computer or other data processing device. The computer executableinstructions may be stored on one or more computer readable media suchas a hard disk, optical disk, removable storage media, solid statememory, RAM, etc. As will be appreciated by one of skill in the art, thefunctionality of the program modules may be combined or distributed asdesired in various embodiments. In addition, the functionality may beembodied in whole or in part in firmware or hardware equivalents such asintegrated circuits, field programmable gate arrays (FPGA), and thelike. Particular data structures may be used to more effectivelyimplement one or more aspects of the disclosure, and such datastructures are contemplated within the scope of computer executableinstructions and computer-usable data described herein.

FIG. 3 illustrates an example environment in which one embodiment of thedisclosure may be employed. This example environment comprises room 300which may be, e.g., a room in user's 306 home. In other embodiments,room 300 may be any space where user 306 may access or consume content.For example, room 300 may be, e.g., user's 306 place of employment, atrain, an airplane, user's 306 school, user's 306 church, etc. Room 300includes a user 306 accessing content via display device 302. Forexample, the user 306 may be watching a television program on displaydevice 302. Display device 302 may be any device in the art suitable fordisplaying content as discussed above in reference to FIG. 1 including,but not limited to, a television, a video display, a computer monitor, adesktop computer, a laptop computer, a tablet computer, a smartphone,and the like. In FIG. 3, display device 302 is located across from user306 sitting on sofa 314. Room 300 also includes further exemplary itemsthat may be found in and around user 306 accessing content, including atable 316, a food or beverage related item such as a cola can 310, andan image such as a picture frame 308. Again, each item is merelyrepresentative of objects which may be located around user 306 accessingcontent and is not intended to limit or define a typical environment foruse of the present disclosure. The range and number of objects which maybe around user 306 in other settings is virtually limitless.

An image sensing or capturing device, such as a camera 304 may beassociated with the display device 302. In FIG. 3, camera 304 isdepicted as sitting to the side of display device 302, but camera 304may be located at any suitable location throughout room 300. Further, insome embodiments, camera 304 may be integral to display device 302. Forexample, televisions, personal computers, laptop computers, tabletcomputers, smartphones, and others may contain cameras integrally formedwithin the display device. The exact location, configuration, andassociation of camera 304 with display device 302 can vary significantlywithout departing from the present disclosure.

Broken lines 318 represent of an exemplary field of view of camera 304.In the illustrated example, camera 304 is positioned such that table316, cola can 310, sofa 314, user 306, and frame 308 are all in itsfield of view. In various other embodiments, camera 304 may bepositioned in such a way that less than all of the items in room 300 areits field of view. For example, in one embodiment camera 304 may befocused on sofa 314, and accordingly only users and/or items situated onor around sofa 314 would be in camera's 304 field of view. In anotherembodiment, camera 304 may be focused on table 316, and accordingly onlyusers and/or items situated near table 316 would be in camera's 304field of view. In still another embodiment, camera 304 may be locatedbehind user 306 and focused towards display device 302, and accordinglyonly users and/or items situated between camera 304 and display device302 would be in camera's 304 field of view. The image capture device(e.g., camera) 304 may also be dynamically controllable, for example, tocontinuously scan the room 300. The location and configuration of camera304 may be further varied without departing from the scope of thisdisclosure.

Broken lines 320 represent an exemplary field of view of user 306. Whileaccessing content, user 306 may alter her field of view depending on,e.g., an object of interest appearing on the display device 302. Asdepicted by broken lines 320, in this example user 306 is focusing hergaze towards the left side of display device 302. As will be appreciatedwith discussion of the following figures, user's 306 field of view isalso focused on the lower part of the screen. That is, in this example,user 306 is focusing her gaze on the lower, left side of display device302.

FIG. 4 illustrates an exemplary captured image 400 by camera 304according to one embodiment of the disclosure. Specifically, capturedimage 400 is representative of an image camera 304 may capture having afield of view illustrated by broken lines 318. In this embodiment,captured image 400 is a perspective view of room 300 and each of itscomponents which appear within broken lines 318. Specifically, thecaptured image 400 includes a perspective view of user 306, table 316,cola can 310, sofa 314, and frame 308. In some embodiments, capturedimage 400 may include one or more objects. As used herein, “object”refers broadly to any identifiable image, item, detail, article, and thelike. For example, in this embodiment, captured image contains brandname object 410 (i.e., the brand name on cola can 310), logo object 404(i.e., the logo on user's 306 hat), graphic object 406 (i.e., thegraphic on user's 306 shirt), and photograph object 408 (i.e., thephotograph contained in frame 308). An image capture device (e.g.,camera 304) or other data capture device may also recognize user's eyes412, and more specifically, what is being viewed by user's eyes 412 aswill be discussed more fully. Again, each object presented in FIG. 4 isfor illustrative purposes only. In other embodiments, user 306 accessingcontent on a display device 302 may have any number of objects on oraround her that may appear in a captured image 400. Objects may berecognized by comparing the detected image with predefined images ofknown objects, logos, brands, words, etc. using any desiredimage-matching technique. Additionally, objects may be recognized asfitting the basic shape of a known type of object (e.g., a shape of asoda can is recognized, even if the logo on the can is not).

As discussed above, user's eyes 412 are generally fixed down and towardsthe left side of display device 302. That is, as seen in the perspectiveview of user 306, user's eyes 412 appear to be gazing down and towardsthe right side of captured image 400 (corresponding to the left side ofdisplay device 302). As will become evident with respect to thediscussion of other figures below, user's eyes 412 are thus anindication that user 306 is viewing an item appearing on the lower, leftside of display device 302.

Moving to FIG. 5, object recognition according to one aspect of thedisclosure is depicted. Broken lines 504, 506, 508, 510, and 512 arerepresentations of one embodiment of the present disclosure recognizingobjects/eyes in captured image 400. Specifically, broken lines 504represent recognition of logo object 404, broken lines 506 representrecognition of graphic object 406, broken lines 508 representrecognition of photograph object 408, broken lines 510 representrecognition of brand name object 410, and broken lines 512 representrecognition of user's eyes 412. Any desired image-matching andrecognition technique may be used to recognize the objects. In oneembodiment, such techniques may be used to, e.g., recognize an objectassociated with an item in captured image 400 (e.g., recognize brandname object 410). In other embodiments, these techniques may be used to,e.g., recognize features of the user 306 herself (e.g., facialrecognition and/or recognize user's eyes 412).

Merely by way of example, recognition may be performed by a processoroperatively coupled to memory containing application software well knownin the art as suitable for object recognition. A processor and/or memoryfor performing such a task may be integral to the display device 302and/or camera 304 or may be remote from each component (at, e.g., acentralized server). This application software may perform variousrecognition tasks to, e.g., captured image 400 in order to recognizeobjects therein. For example, the application software may comprise adatabase of exemplar objects and a processor may thus compare, e.g.,captured image 400 to the database of exemplar objects. If the capturedimage 400, or any aspect of the captured image 400 (e.g., a specifictext string, logo, unique design, color pattern, etc.) is similar to anexemplar image, the processor running the application software mayrecognize the captured image 400 and/or specific aspect of the capturedimage 400 as being the same as the exemplar object. This method ofobject recognition is merely illustrative of one method of recognizingobjects in a captured image, and is not intended to define or limit theobject recognition process of the present disclosure. Those skilled inthe art will appreciate various other methods and hardware/softwarecombinations that are suitable for recognizing objects within a capturedimage. Any number of these suitable methods for object recognition maybe employed herein without departing from the present disclosure.

FIGS. 6a-6c illustrate embodiments of displaying additional content to auser once objects have been recognized in the captured image 400.Specifically, FIG. 6a illustrates example content 602 that may bedisplayed on display device 302 associated with camera 304. By way ofexample, in FIG. 6a content 602 is a video presentation of a footballgame. The video images in content 602 may further include objectsappearing in the video image. The objects appearing in the video imagemay include, e.g., discrete products, logos, signs, actors, etc., whichuser's eyes 412 may focus on when viewing content 602. Again, by way ofexample, in FIG. 6a , content 602 comprises objects includingadvertising sign 608, depicting “ABC Cola,” and spectator 612, wearing ashirt depicting “xYz Cola.” These objects are merely exemplary of typesof objects that may be associated with a broadcast of a football game.In other embodiments, objects in content 602 may include, e.g., aproduct placed in a television show (such as a cola can sitting on atable or an automobile driving down the street), text associated withthe content (such as a phone number or webpage address), or any otherrecognizable image or part of the video image. Returning to FIG. 6a , inthis embodiment display device 302 may be, e.g., a television set,desktop computer, laptop computer, tablet computer, or smartphone andthe like, displaying a live football game via fiber, coaxial cable,wireless, or other transmission media. Further, camera 304 may beintegral to display device 302, or may be located externally to displaydevice 302 and operatively coupled to display device by any method wellknown in the art.

FIG. 6b illustrates a first embodiment of displaying additional contenton display device 302 according to one aspect of the disclosure. In FIG.6b , display device 302 is displaying content 602 (in this example, afootball game). Display device 302 is further displaying additionalcontent 606. Specifically, the recognized brand name object 410 (which,as noted above with respect to FIG. 4, was identified in the room withthe user) is processed by processor 604, as illustrated by processingcloud 610. The processing clouds used throughout the disclosure are forillustrative purposes only and are not intended to limit or define themanner in which objects are processed.

Processor 604 may be a processor integral to camera 304 and/or displaydevice 302. Alternatively, processor 604 may be a processor located inanother of a user's devices which are operatively coupled to orotherwise communicatively linked to camera 304 and display device 302.For example, processor 604 may be located in a desktop computer, laptopcomputer, tablet computer, smartphone, and the like operatively coupledto camera 304. Alternatively, processor 604 may be located remotely fromdisplay device 302 and camera 304. For example, processor 604 may belocated at a centralized server. Referring back to FIG. 1, in oneembodiment processor 604 may be located at a server in local office 103.In this embodiment, the display device 302 and camera 304 may becommunicatively linked to local office 103 through, e.g., links 101.Processor 604 may be located at push notification server 105, contentserver 106, and/or application server 107. Processor 604 may take avariety of forms in a variety of locations without departing from thedisclosure.

Returning to FIG. 6b , recognized brand name object 410 is processed byprocessor 604. In one embodiment, processing by processor 604 includesdetermining whether the recognized object 410 is associated with anyadditional content that can be provided to the user. For example, in theimage processing that resulted in identifying the object 410 as being an“ABC” brand soda can, the processing may include determining whether the“ABC” brand soda has been associated with additional content relating tothe brand. The additional content may be, for example, a pop-upadvertisement for the product, a coupon or discount offer, or any otherdesired additional content. The determination may be made by consultinga target list or database that identifies relationships between visualimages, content sources, and additional associated content. The list maybe stored locally (e.g., on memory operatively coupled to processor 604)or remotely (e.g., at local office 103 and communicatively coupled toprocessor 604 by link 101).

For example, in FIG. 6b , brand name object 410 contains the brand name“ABC Cola.” In this embodiment, processor 604 may compare “ABC Cola” tothe target list, and identify an entry associating the content sourcefor “ABC Cola”. If the target list contains an entry for “ABC Cola,”there may be additional content associated with the target object. Forexample, the makers of “ABC Cola” may identify additional images to beshown to the user, providing additional information or offers regardingtheir brand of soda. The additional information may include a resourceaddress or link to additional content. In some embodiments, even acompetitor of ABC Cola (e.g., “xYz Cola”) may associate content with theobject “ABC Cola.” This content may be, e.g., an advertisement or othercontent associated with xYz Cola to be sent to a known user of ABC Colain an attempt to have that user try their product. In this embodiment,because the xYz Cola content is associated with ABC Cola, when theprocessor 604 processes brand name object 410, determining a targetmatch, processor 604 can send the additional content 606 to displaydevice 302.

Display device 302 may display the additional content 606. In thisexample, additional content 606 is an advertisement for a competingcompany (i.e., xYz Cola) to the company associated with brand nameobject 410 which appeared in captured image 400. Thus, in this example,when camera 304 captures an image of brand name object 410 in the roomwith the user viewing the football game, an advertisement for acompeting company can be returned and overlaid on the display device. Inthis way, advertisers may target certain users for advertisements ratherthan sending out a “one-size-fits-all” advertisement. Specifically,advertisers may target those who patronize a similar, yet competing,product (i.e., ABC Cola) in an attempt to have them try their product(i.e., xYz Cola).

Additional content 606 may take a variety of forms well known in the artwithout departing from the present disclosure. For example, in FIG. 6b ,additional content 606 appears as a banner overlaid on content 602.Alternatively, as is well known in the art, additional content may be apopup window rather than a banner advertisement. In this embodiment,additional content 606 may be a window providing a message oradvertisement that overlays content 602 until, e.g., a user closes thepopup window. Additionally, additional content 606 may take the form of,e.g., a full-page advertisement. In this embodiment, additional contentmay overlay, and thus cover up, all of content 602. Alternatively,additional content may be, e.g., a television commercial, and thusdelivered to user 306 at a scheduled commercial break. Alternatively, aswill be appreciated by those skilled in the art, display device 302 mayhave multiple feeds. One feed may carry, e.g., content 602, whileanother feed may go to, e.g., a quadrature amplitude modulation (QAM)delivery for video-on-demand (VOD) streams. In such an embodiment,additional content 606 may comprise a channel change to the VOD streamcarrying, e.g., an advertisement. In some embodiments, the additionalcontent 606 may be an Internet address that the viewer can access fromthe display device 302 or via a different device. Additional content 606may take any number of forms well known in the art without departingfrom the instant disclosure.

Those skilled in the art will also recognize that additional content 606need not be associated with a competing company to that of a matchedobject. As presented in the various exemplary embodiments below, thecontent may be complimentary (i.e., associated with the same or relatedentity) or neutral (i.e., associated with an entity that is merelyinterested in reaching out to, e.g., soda drinkers). For example, ABCCola, rather than xYz Cola, may associate additional content 606 withobjects, such as brand name object 410, which depict their own brandname. That is, when the processor 604 processes brand name object 410,it may match the object on the target list to content provided by ABCCola (rather than xYz Cola). Processor 604 may then return additionalcontent 606 to display device 302 which is complementary to the brandname object 410. In this embodiment, additional content 606 may depict,e.g., a loyalty program associated with ABC Cola. That is, entities mayuse an embodiment of the present disclosure to, e.g., track user's 306use of their products. The user 306 may then be rewarded when images arecaptured of him or her using a certain product, or, as explained indetail below, when user 306 views an object on display device 302related to a certain product. Specifically, returning to FIG. 6b , user306, who appears with brand name object 410 in captured image 400, couldbe rewarded for having Cola Can 310 around while accessing content. Forexample, user 306 could receive, e.g., loyalty points (deposited in auser account as will be discussed below) when captured image 400contains objects associated with ABC Cola or when user 306 views anobject associated with ABC Cola. In such an embodiment, instead of acompeting advertisement, additional content 606 may contain anotification that these loyalty points have been deposited in user's 306account. In this way, user 306 would be encouraged to continue topatronize a certain brand because he or she will be rewarded when imagesare captured including the brand name object 410. This exampleillustrates how one embodiment of the present disclosure providesadvantages to advertisers, as compared to traditional contentdistribution techniques, because advertisers can target currentcustomers and strengthen their consumer base, while refraining fromproviding content to, e.g., users of another product or users who maynot drink soda.

As noted above, an object may be recognized specifically, e.g., a can ofABC brand soda. Alternatively, an object may be recognized as a generalobject, but the actual source may be unrecognized by the imageprocessor. For example, the camera 304 may be configured to recognizecans in general, without recognizing an associated brand or label on thecan. FIG. 6c represents one embodiment wherein a recognized object doesnot match a target object on the target list. That is, even though anobject may be recognized as a general object using any of theaforementioned object recognition techniques (e.g., the general shape ofa can), there may not be a match with any corresponding content sourcestored on the target list. In such a case, an embodiment of the presentdisclosure may obtain further information about the recognized objectand, if ultimately identified, store the recognized object in the targetlist for future matches. Specifically, in the embodiment depicted inFIG. 6c , the captured object may be displayed to the user with arequest that the user enter information about a recognized object. Here,processor 604 processes photograph object 408, as represented byprocessing cloud 614, but processor 604 does not match photograph object408 to a brand-specific target object on the target list. In thisexample, additional content 616 is returned to display device 302, whichprompts user 306 for more information regarding photograph object 408.Specifically, a representation 618 of photograph object 408 is presentedvia additional content 616 on display screen 302. In this embodiment,user 306 is prompted to enter more information regarding photographobject 408. For example, user 306 may be prompted to enter the name orbrief description of photograph object 408. In the depicted example,user 306 may input information such as “mountains” or “Denver, Colo.”

Further, one embodiment may then use this information to storephotograph object 408 on the target list for future matches.Specifically, processor 604 may transmit the information and/or arepresentation 618 of photograph object 408 to a remote computing device(not pictured) which identifies the object. Remote computing device maycomprise an object database containing an object that matches photographobject 408. Alternatively, processor 604 and/or a remote computingdevice may perform, e.g., an internet search using the inputtedinformation and/or representation 618 of photograph object 408 toidentify the object. If successfully identified, one embodiment of thedisclosure may then store the image, e.g., representation 618, on thetarget list. Such an embodiment may result in an ever-evolving andgrowing list of target objects. Interested entities may periodicallyreview such a list, and associate additional content with newlyidentified target objects so that for future matches processor 604 willreturn additional content when, e.g., photograph object 408 isrecognized.

FIG. 7 presents an embodiment of the disclosure wherein more than onerecognized object from captured image 400 is used to provide additionalcontent. Specifically, in FIG. 7, logo object 404, graphic object 406,photograph object 408, and brand name object 410 are all identified inthe room of the viewer and processed by processor 604, as illustrated byprocessing cloud 702. In this embodiment, each object may be associatedwith a target object on the target list. That is, when each object isprocessed by processor 604, processor 604 may return multiple matches,and each of the multiple matches may be associated with differentadditional content. Alternatively, less than all of objects may beassociated a target object on the target list.

In the embodiment depicted, processor 604 may select additional contentto display on display device 302 based on the number of instances amatched object appears in the captured image 400. For example, in thisembodiment, processor 604 processes, as represented by processing cloud702, brand name object 410, logo object 404, graphic object 406, andphotograph object 408. Two of those objects, namely brand name object410 and logo object 404, are associated with ABC Cola. The other objectsare unrelated to each other. In this embodiment, when more than oneobject matches a target object (or content source) in the target list,the additional content 704 selected for display may be the one thatcorresponds with the source or target object having the most matches.That is, additional content 704 is associated with brand name object 410and logo object 504, which both match an “ABC Cola” target object.Because brand name object 410 and logo object 404 both match the sametarget object, and no other target objects had more matching objects inthe captured image, processor 604 will return additional content 704,associated with the target object matching brand name object 410 andlogo object 404. In this embodiment, additional content 704 is a fullpage advertisement depicting a competing company to the matched targetobject; namely “xYz Cola.” As discussed above, additional content 704may take a variety of forms (e.g., banner, popup, full-page, etc.) andbe associated with a variety of entities (e.g., competing,complimentary, neutral) without departing from the scope of thisdisclosure.

In an alternative embodiment, processor 604 may return additionalcontent 704 associated with a recognized object appearing the leastamount of instances in the captured image. For example, in analternative embodiment there may be two recognized objects associatedwith ABC Cola (e.g., brand name object 410 and logo object 404) and tworecognized objects associated with Denver, Colo. (e.g., photographobject 408 and an additional object not pictured). In this embodiment,processor 604 may return additional content 704 associated with, e.g.,graphic object 406. That is, in this embodiment objects associated withState University (e.g., graphic object 406) appear only once whileobjects associated with other target objects (namely, ABC Cola andDenver, Colo.) appear more than once, and thus processor 604 returnsadditional content 704 associated with the object appearing the leastamount of instances in captured image 400.

Moving to FIG. 8, one embodiment of the present disclosure isillustrated wherein additional content 804 is delivered based on acombination of recognized objects. Specifically, display device 302associated with camera 304 from room 300 is once again depicted.However, in this example, multiple recognized objects from capturedimage 400 are processed and a combination of the matched objects is usedto determine additional content 804. Specifically, in this embodiment,processor 604 processes both graphic object 406 and photograph object408 as illustrated by processing cloud 802 and returns additionalcontent 804. Here, additional content 804 is associated with more thanone of the recognized objects; namely both graphic object 406 andphotograph object 408. For example, processor 604 may determine graphicobject 406 is associated with State University. Processor 604 mayfurther determine photograph object 408 is associated with Denver, Colo.Processor unit 604 may then return additional content 804 associatedwith both State University and Denver, Colo. That is, processor 604 may,e.g., match multiple recognized objects (e.g., graphic object 406 andphotograph object 408) to common additional content on the target list(e.g., additional content 804). In this embodiment, when the capturedimage 400 contains multiple objects, the processor 604 returnsadditional content 804 which corresponds to a combination of more thanone of the recognized objects. In the example depicted, processor 604returns additional content 804 associated with both graphic object 406(associated with State University) and photograph object 408 (associatedwith Denver, Colo.); namely, an advertisement for tickets to an athleticevent involving State University in Denver, Colo. In this embodiment,multiple target objects on the target list may be associated with commonadditional content 804, and the processor may return additional content804 when it determines more than one recognized object matches more thanone target objects associated with common additional content 804.

The discussion above has addressed recognition of objects that are inthe room with the viewer. In alternate embodiments, the present systemcan recognize the user's attention (e.g., the focus of their eyes), anddeliver additional content based on areas of interest to the user. FIGS.9a-9b illustrates one embodiment of the present disclosure whereinprocessor 604 returns additional content 904 according to objects towhich user 306 is paying special attention. Specifically, user 306 isviewing content 602 on display device 302. Content 602 may contain oneor more objects. For example, content 602 contains advertising sign 608and spectator 612. Advertising sign 608 may be, e.g., a sign containingan advertisement placed along the side of a field at a major sportingevent as is common in advertising. Spectator 612 may be, e.g., a persondepicted on content 602 who has a logo or other identification on herclothing. In the example depicted in FIG. 9a , advertising sign 608 isassociated with “ABC Cola,” and spectator 612 is associated with “xYzCola.” It should be noted, as throughout the disclosure, the particulartype and substance of the objects found in content 602 may vary greatlybetween various embodiments without departing from the presentdisclosure. Advertising sign 608 and spectator 612 are depicted forillustrative purposes only.

In FIG. 9a , user 306 is viewing advertising sign 608 as is depicted bybroken lines 320. That is, user's 306 field of view is generally focusedto the lower, left side of display device 302, the general locationwhere advertising sign 608 is located. In FIG. 9b , additional content904 is delivered based on processor 604 determining user 306 is viewingadvertising sign 608. A data capture device captures data related towhat is being viewed by user's eyes 412. For example, in the depictedembodiment, camera 304 captures image 400, and user's eyes 412 arerecognized using any desired eye tracking technique. The captured datais then processed to determine what the user is viewing. In thisembodiment, processor 604 processes user's eyes 412, as illustrated byprocessing cloud 902, to determine, e.g., that user 306 is generallyviewing content in the lower, left side of display device 302, and thatthe user's attention has remained for longer than a predeterminedminimum amount of time (e.g., 1 second). Processor 604 furtherdetermines that advertising sign 608 is located in the lower, left sideof display device 302. Processor 604 may use a variety of methods todetermine what object is located in the lower, left side of displaydevice 302. For example, content 602 may be accompanied by information(e.g., metadata, manifest file, etc.) from the content provider orothers, which instructs processor 604 as to the type and location ofobjects contained within content 602. In one embodiment, the content maybe provided with a script indicating the type and location of eachobject on the displayed video image. Such a script may, e.g., divide thedisplayed image into discrete portions, and indicate which objects arelocated in each portion of the displayed image. An exemplary script maydivide the content into four quadrants and indicate what objects appearin each quadrant. In the depicted example, the script may indicate thatan “ABC” object is located in the lower, left quadrant, and an “xYz”object is located in a lower, right quadrant. This form of roughapproximation of the onscreen location of objects of interest may assistthe eye-tracking process, reducing the degree of accuracy required forthe eye tracking. Alternatively, processor 604 may capture an image(e.g., a screen shot) of content 602 at substantially the same momentcaptured image 400 is captured, and use any of the aforementioned objectrecognition techniques to determine what types of objects, and thelocation thereof, are contained in content 602. Those skilled in the artwill appreciate a variety of other well-known techniques to recognizethe type of location of objects associated with content 602 withoutdeparting from the scope of this disclosure.

Returning to FIG. 9b , once processor 604 has determined that user 306is viewing the lower, left side of display device 302, and thatadvertising sign 608 is located at the lower, left side of displaydevice 302, processor 604 may return additional content 904 to displaydevice which is associated with advertising sign 608. Specifically, inone embodiment, processor may compare the object associated withadvertising sign 608 (i.e., “ABC Cola”) with a target list of targetobjects. If processor 604 determines, e.g., “ABC Cola” matches a targetobject on the target list, processor 604 then determines if additionalcontent is associated with the target object. In this example,additional content 904 is associated with “ABC Cola.” Specifically, acompeting product (i.e., “xYz Cola”) has associated an advertisementwith “ABC Cola.” Thus, when processor 604 determines user 306 is viewingadvertising sign 608 (as opposed to, e.g., spectator 612), it may returnadditional content 904. In this way, entities such as, e.g.,advertisers, may target users depending on content they are interactingwith. Specifically, entities may determine what objects users areviewing and deliver additional content accordingly.

Those skilled in the art will appreciate the embodiment depicted inFIGS. 9a-9b is not limited to advertisers of competing products. Forexample, in another embodiment, a content provider may charge an entity,e.g., an advertiser, depending on the number of views and/or the amountof time users view subcontent delivered with content 602. Increasingly,product placement is becoming more common in content due to, e.g., theadvent of digital video recorders (DVR) and other technologies whichallow user 306 to fast-forward or otherwise skip commercials. Therefore,data regarding whether user 306 viewed certain objects may be desirableeven if additional content 904 is not associated with that object. Forexample, returning to FIG. 9a , content 602 includes advertising sign608, which depicts “ABC Cola,” and spectator 612, which depicts “xYzCola.” In one embodiment of the present disclosure, user's eyes 412,and/or multiple users' eyes in the aggregate, could be used to determinehow much a particular entity owes for advertising. Specifically, acontent provider of content 602 may track the number of instances user306 looks at advertising sign 608 and/or the total amount of time user306 spends looking at advertising sign 608, and charge ABC Cola a feeaccordingly. Additionally, a content provider of content 602 may trackthe number of instances user 306 looks at spectator 612 and/or the totalamount of time user 306 spends looking at spectator 612 and, and chargexYz Cola a fee accordingly.

In some situations, processor 604 may determine multiple objects in aroom and/or detect that user 306 has viewed multiple objects in content602 which are associated with competing entities. For example, processor604 may determine room 300 contains objects associated with both ABCCola and xYz Cola, or that user 306 has viewed objects associated withboth ABC Cola and xYz Cola. In such a situation, processor 604 mayreturn additional content corresponding to each of the detectedobjects/objects. For example, user 306 may be provided with commercialsfor a variety of products (e.g., one commercial for ABC Cola and one forxYz Cola).

However, entities (e.g., advertisers) may not wish to share theiradvertising space with other advertisers, particularly with competitors.Thus, in some embodiments, processor 604 may not display additionalcontent associated with certain recognized objects and/or viewedobjects. For example, entities may only wish to provide additionalcontent to viewers if they are guaranteed their advertisement will beexclusive; i.e., if the objects associated with their additional contentare the only recognized objects and/or if they are guaranteed otherentities' additional content will not be displayed. In such embodiments,processor 604 may alter, or rather customize, the content 602 beingviewed to provide an entity with exclusive advertising space. Forexample, an entity associated with advertising sign 608 (e.g., ABC Cola)may agree to pay a content provider based on, e.g., a number of viewsadvertising sign 608 receives only if they are guaranteed an exclusivespot within content 602. In such an embodiment, processor 604 may blockout the logo on spectator 612 (e.g., scramble the “xYz” logo on theshirt) and/or replace the logo at spectator 612 with graphical overlayof a logo depicting ABC Cola. In such an embodiment, an entity may pay acontent provider for the rights to have their additional contentpreferred over, e.g., a competitor's additional content when objectsassociated with both entities are recognized.

In other embodiments, a content provider may provide content withgeneric product placements and replace such generic products withadditional content according to any of the techniques discussed herein.For example, in some embodiments a content provider may provide content602 with one or more objects (such as advertising sign 608 or spectator612) that have either generic logos or no brand indicia associated withthem. Processor 604 may then replace the generic object with abrand-specific object in response to any of the aforementioned objectrecognition techniques. For example, in one embodiment, processor 604may recognize certain objects within a room and customize objects incontent 602 accordingly (i.e., provide additional content as a graphicaloverlay on a generic product, such as through use of digital imagereplacement and graphical editing). In other embodiments, processor 604may determine what objects user 306 views at one point in the content602 and alter other generic objects in the content 602 accordingly. Inthis way, entities, such as advertisers, can reach their target audienceusing the teachings of this disclosure even if, e.g., user 306 is usingDVR or technology and thus normally does not access traditionalcommercial spots by customizing generic products placed within deliveredcontent.

In other embodiments, an entity may track user's 306 interest in objectsfor a variety of reasons. For example, ABC Cola may track user's 306interest in objects, such as advertising sign 608, and compare this to,e.g., other advertisements depicting ABC Cola and/or other objectsdepicting other products (e.g., spectator 612) to count the amount ofattention being given to each advertisement, and determine whichadvertising campaign is more effective. Alternatively, an entity suchas, e.g., ABC Cola, may use an embodiment of the present disclosure inconnection with a loyalty program. That is, ABC Cola may track user's306 interest in advertising sign 608 and subsequently reward a user whoviews ABC Cola content on display device 302 more than a minimum numberof times, or at a minimum frequency.

Similarly, one embodiment of the present disclosure may use furtherfacial recognition techniques. For example, in addition to recognizingan object associated with user 306 accessing content and/or user's eyes412 to determine which objects user 306 is viewing on display screen302, one embodiment of the disclosure may use facial recognition todetermine, e.g., an age of user 306. That is, an entity may associatevarious additional content with a target object on a target list, andmay wish to tailor which of the various additional content it returnsaccording to the user's 306 approximate age. In such an example, oneembodiment of the disclosure may use facial recognition software todetermine, e.g., which of a known plurality of users is accessingcontent (or present in the room). In such an embodiment, each registereduser may have a stored image of herself in, e.g., a user profile(discussed more fully below) and the embodiment may recognize whichuser(s) is accessing content by capturing an image of the faces of thoseaccessing content and comparing the captured faces to the stored image.Once determined, additional content can be tailored according to auser's age and/or stored preferences. In another embodiment, facialrecognition may be used to approximate the age of user(s) accessingcontent using well-known facial recognition techniques (which will notbe described at length here). In this embodiment, there may be more thanone additional content associated with a target object, and theprocessor 604 will select which to return according to the user's age.

FIG. 10 is an example captured image 1000 which is captured later intime than captured image 400 of FIG. 4. Captured image 1000 is similarto captured image 400; however, in captured image 1000 there is adifferent type of cola on table 316. Specifically, cola can 1002 nowsits atop table 316, which has the brand name “xYz Cola.” As depicted bybroken lines 1010, the illustrated embodiment recognizes brand nameobject 1004, corresponding to the “xYz Cola” brand name on cola can1002. The embodiment may further recognize other objects in the capturedimage as well. For example, the embodiment depicted also recognizesgraphic object 406, as represented by broken lines 1006, photographobject 408, as represented by broken lines 1008, and user's eyes 412, asrepresented by broken lines 1012. Returning to FIG. 5, three of thesefour recognized objects are the same as those recognized in capturedimage 400. Specifically, graphic object 406, photograph object 408, anduser's eyes 412 were each recognized in captured image 400. However,brand name object 1004, which depicts “xYz Cola,” did not appear, wasthus not recognized, in captured image 400. In some embodiments, thenewer object may be given a higher priority in determining what theadditional content will be. For example, a weighting scale may bedefined, with higher weights being given to objects that are new, or thescale may gradually reduce weights of objects as the object remains insubsequent images. Further, brand name object 410 and logo object 404,which were recognized in captured image 400 and depicted “ABC Cola,” nolonger appear in captured image 1000. The absence of objects from oneimage to the next may also be used. For example, the processor may beconfigured to note the absence of the soda can, and determine theduration of the object in the captured image (e.g., the can appeared inimages captured over a 20-minute span), and that information can berelayed to the soda supplier for research purposes.

FIG. 11 illustrates an embodiment of the disclosure in which processor604 determines there has been a change in captured objects over time andreturns additional content 1106 accordingly. Namely, in this embodiment,processor 604 compares the objects recognized in captured image 1000with the objects earlier recognized in captured image 400. Asillustrated by overlapping processing clouds 1102 and 1104, processor604 determines there has been a change in the recognized objects in eachcaptured image 400 and 1000. Specifically, processor 604 may recognizethat graphic object 406, photograph object 408, and user's eyes 412appear in both captured images. However, processor 604 may alsorecognize that brand name object 410, which appeared in captured image400 and depicted “ABC Cola,” and logo object 404, which appeared incaptured image 400 and also depicted “ABC Cola,” do not appear incaptured image 1000, and that brand name object 1004, which appears incaptured image 1000 and depicts “xYz Cola,” did not appear in capturedimage 400. Accordingly, processor 604 may return additional content1106. That is, in one embodiment, additional content may be associatedin the target list with a change from objects depicting “ABC Cola” toobjects depicting “xYz Cola.” In this example, additional content 1106is an advertisement by ABC Cola. In this way, entities can target usersbased upon their habits over time. In the example presented in FIG. 11,ABC Cola is targeting users who have potentially switched to xYz Colafor delivery of additional content 1106. Specifically, when processor604 determines a switch from recognized objects associated with ABC Cola(e.g., brand name object 410 and logo object 404) to recognized objectsassociated with xYz Cola (e.g., brand name object 1004), processor 604returns additional content (e.g., additional content 1106) in an attemptto have the associated user return to ABC Cola. Those skilled in the artwill recognize such techniques are not limited to use by entitiesassociated with the first recognized object. For example, in analternative embodiment, xYz Cola may associate content in the targetlist with a change in recognized objects from ABC Cola to xYz Cola. Inthis example, xYz Cola can thus, e.g., reward or otherwise thank a userfor making the switch.

In another embodiment, a change in user's 306 interest in objectsappearing on the television or display screen may be used to returnadditional content 1106. Specifically, one embodiment may process achange in the objects user 306 views on display device 302. Returningto, e.g., FIGS. 9a and 9b , if, at a later time, processor 604determines, by processing data related to what is being viewed by user'seyes 412, that user 306 is now viewing object associated with, e.g., xYzCola frequently, processor 604 may process the change (i.e., the changeform viewing advertising sign 608 to viewing xYz objects) and returnadditional content 1106 accordingly. For example, ABC Cola may targetusers who once were interested in ABC Cola objects (e.g., advertisingsign 608) but who now often view xYz Cola objects.

FIG. 12 illustrates one embodiment of the present disclosure whereinmultiple objects associated with multiple display devices and multipleusers are processed to display additional content 1240 on display device1202. Specifically, in this embodiment, display devices 1202, 1204,1206, and 1208 are each associated with a different user, andcommunicatively coupled to a centralized server 1218. Centralized server1218 may take, e.g., any of the various forms as discussed above. Forexample, centralized server may be located at local office 103. Further,centralized server 1218 may be push notification server 105, contentserver 106, and/or application server 107. Those skilled in the art willappreciate various embodiments of centralized server 1218 withoutdeparting from the scope of the disclosure.

Each display device 1202, 1204, 1206, and 1208 has an image or datacapture device associated with it, such as, e.g., camera 1210, 1212,1214, and 1216, respectively. Each camera may capture an image of a userand her surroundings while accessing content and recognize at least oneobject in the captured image, represented in FIG. 12 by recognizedobjects 1220, 1222, 1224, 1226, and 1228. Centralized server 1218 maythen process each of recognized objects 1220, 1222, 1224, 1226, and1228, and return additional content to at least one of the displaydevices 1202, 1204, 1206, or 1208.

In the example depicted in FIG. 12, a user is accessing content 1238 viadisplay device 1202. In this example, content 1238 is a football gamewhich includes objects advertising sign 1242 and spectator 1244. Theuser may have several items surrounding her while accessing the content1238, and the camera 1210 may capture an image of the user accessing thecontent. In this example, the camera 1210 has captured and recognized,using any object recognition method well known in the art and/ordescribed above, logo object 1220 and brand name object 1222. Asdescribed above, an image or data capture device may also capture datarelated to objects being viewed by user's eyes (not pictured). In thedepicted example, cameras 1212, 1214, and 1216, have captured andrecognized brand name object 1214, logo object 1226, and logo object1228, respectively associated with other users accessing content atdisplay devices 1204, 1206, and 1208. Again, cameras 1212, 1214, and1216 or other data capture devices may also capture and recognize theuser's eyes associated with each respective user accessing content. Forsimplicity, in this example only objects associated with ABC Cola areillustrated, but it should be well understood that cameras 1210, 1212,1214, and 1216 may capture and recognize other objects besides objectsassociated with ABC Cola. In any event, centralized server 1218determines that camera 1210, associated with the user accessing contentat display device 1202, has captured more ABC Cola objects than any ofthe other cameras 1212, 1214, and 1216. Additional content 1240 is thenreturned to display device 1202. In this example, additional content1240 announces to the user at display device 1202 that she is thebiggest ABC fan in the neighborhood and offers her a free six-pack ofcola. In one embodiment, an entity can thus promote their product by,e.g., establishing a competition among those accessing content andreward the user(s) who supports the company the most.

In the example depicted in FIG. 12, the number of users communicativelycoupled to one another by centralized server 1218 may vary. For example,in the embodiment depicted in FIG. 12, the users are associated as beingwithin the same geographic area; namely, the same neighborhood.Alternatively, the users could be from, e.g., a common workplace,school, church, or even a user-defined group associated with, e.g.,social media. For example, instead of comparing the number of instancesof recognized objects from one user to those of others in that user'sneighborhood, the centralized server 1218 may compare the number ofinstances recognized objects from one user with, e.g., others as definedby the user's friend list on a social media website.

In an alternative embodiment, additional content 1240 may be retunedaccording to the objects of each user is viewing on each respectivedisplay device. Similar to the method as presented above with respect toFIGS. 9a -9 b, centralized server 1218 may process data related to whatis being viewed by each user's eyes to determine which object, if any,each user is viewing. For example, in the embodiment depicted in FIG.12, content 1238 comprises objects including advertising sign 1242 andspectator 1244. Centralized server 1218 may determine, e.g., that a userat display device 1202 views advertising sign 1242 more times than theviewers at display devices 1204, 1206, and 1208. Alternatively,centralized server 1218 may determine, e.g., that a user at displaydevice 1202 views advertising sign 1242 for a longer duration than otherviewers. The centralized server 1218 may thus return additional content1540 based on the viewing patterns of each user. Here, a user at displaydevice 1202 is being rewarded as “the biggest ABC Fan” because she,e.g., viewed advertising sign more times and/or longer than the otherusers.

FIG. 13 illustrates an embodiment of the disclosure wherein additionalcontent is delivered to more than one display device according torecognized objects from more than one user. Specifically, displaydevices 1302, 1304, and 1306, associated with cameras 1308, 1310, and1312 respectively, are each communicatively coupled to centralizedserver 1336. As with the above embodiments, centralized server 1336 maybe any desired type of computer server, and may be, e.g., a push server105, content server 106, and/or application server 107 located at localoffice 103. Objects 1320, 1324, 1328, 1332, and 1334, which areassociated with users at each display device 1302, 1304, and 1306 as isunderstood with reference to the above description, are recognized. Aspresented above, recognition may be achieved by any well-known method inthe art and may be performed, e.g., locally at each camera 1308, 1310,and 1312, or remotely such as at, e.g., centralized server 1336.Centralized server 1336 then processes each object 1320, 1324, 1328,1332, and 1334, as illustrated by processing clouds 1314, 1316, and 1318by, e.g., any of the aforementioned methods, and returns additionalcontent 1338 to each display device (depicted at display devices 1302,1304, and 1306 as 1338 a, 1338 b, and 1338 c, respectively). In thisembodiment, processor 1336 sends the same additional content 1338 toeach display device 1302, 1304, and 1306, according to the aggregatematches of processed objects 1320, 1324, 1328, 1332, and 1334. Forexample, processor 1336 may associate objects 1320 and 1324, which bothdepict ABC Cola, to content associated with xYz Cola (e.g., if xYz Colais targeting current ABC Cola customers). However, processor 1336 mayalso associate objects 1328, 1332, and 1334, which each depict xYz Cola,with content associated with ABC Cola (e.g., if ABC Cola is targetingcurrent xYz Cola customers). In the aggregate, processor 1336 thusmatches more objects to ABC Cola content (i.e., objects 1328, 1332, and1334) than to xYz Cola content (i.e., objects 1320 and 1324). In theembodiment depicted in FIG. 13, processor 1336 then sends the sameadditional content 1338 to each display device 1302, 1304, and 1306. Inthis example, additional content 1338 is associated with ABC Cola. Thatis, because, in the aggregate, processor 1336 matched more objects in,e.g., a target list with content associated with ABC Cola than withcontent associated with xYz Cola, each display device receivesadditional content 1338 associated with ABC Cola. Thus, in thisembodiment, an entity may target a large number of users based on, e.g.,a majority preference of those accessing content at a given time. Thisprovides entities with the ability to track their target audience inreal-time, and respond with targeted additional content.

Of course, those skilled in the art will appreciate that the othervarious embodiments discussed above may be incorporated into theexemplary embodiment depicted in FIG. 13. For example, the embodimentdepicted in FIGS. 9a-9b may be used in conjunction with the exampledepicted in FIG. 13. That is, each user, located at display devices1302, 1304, and 1306, may be accessing content 602 which may compriseobjects including, e.g., advertising sign 608 and spectator 612.Further, data capture devices associated with each display device, suchas, e.g., cameras 1308, 1310, and 1312, may capture data related to whatis being viewed by each user's eyes at display devices 1302, 1304, and1306. Using techniques such as, e.g., those described in connection withFIGS. 9a -9 b, the embodiment in FIG. 13 may then determine whichobjects each user is viewing. For example, the embodiment may track howlong each viewer spends viewing, e.g., advertising sign 608, and howmuch time each user spends viewing spectator 612. Alternatively, theembodiment may count how many instances each user views each object. Inany event, centralized server 1336 may then aggregate the information,and determine additional content 1338 to return accordingly. Forexample, in one embodiment centralized server 1336 may determine that onaverage users are viewing spectator 612, associated with xYz Cola, morethan they are viewing advertising sign 608. In this example, ABC Colamay then have content associated with, e.g., xYz Cola, and thuscentralized server may return additional content 1338 depicting ABC Colain an attempt to divert users' interest from xYz Cola and to ABC Cola.Alternatively, xYz Cola may associate additional content 1338 with xYzCola. That is, xYz Cola may be interested when, in the aggregate, usersare viewing objects associated with xYz Cola, and thus follow-up withadditional content 1338 depicting, e.g., xYz Cola to further interest intheir product.

FIG. 14 illustrates an embodiment of the disclosure utilizing more thanone display device. Specifically, FIG. 14 illustrates a room 1400comprising a display device 1402, a camera 1404 associated with thedisplay device, a user 1406 sitting on a sofa 1414, a frame 1408, and acola can 1410 sitting on a table 1416. In this respect, room 1400 andits contents are similar to room 300 as depicted in FIG. 3. However,room 1400 additionally contains second display device 1422. In thisembodiment, second display device is depicted as a laptop computer, but,as with display device 1402, second display device 1422 may be any typeof display device well known in the art and used for accessing content.For example, second display device 1422 may be a television set, desktopcomputer, tablet computer, smartphone, and the like. Broken lines 1418represent a field of view of camera 1404. As discussed with respect toFIG. 3, the exact location and field of view of camera 1404 may varydepending on the particular embodiment of the present disclosure. In theexample depicted in FIG. 14, broken lines 1418 represent a field of viewwhich includes table 1416 with cola can 1410, sofa 1414 with user 1406,frame 1408, and second display device 1422. As is now well understoodwith reference to the previous figures and description, the camera 1404and display device 1402 may capture an image of those items in camera's1404 field of view, and applying, e.g., one of the aforementioned objectrecognition techniques, recognize objects associated with user 1406accessing content on display device 1402. Broken lines 1420 representuser's 1406 field of view. That is, user 1406 may focus on area of thedisplay device 1402 when, e.g., the display device 1402 displays objectsof interest to the user. In the example depicted in FIG. 14, user 1406is viewing content on the left side of display device 1402, asrepresented by broken lines 1420. As is understood with reference to theprevious figures and description, a data capture device, such as, e.g.,camera 1404, may capture data related to what is being viewed by user's1406 eyes, and, in some embodiments, the data may be used to determinewhich, if any, objects on a display device 1402 user 1406 is viewing atany given time. Use of this dual-screen configuration is discussed withrespect to FIG. 15.

FIG. 15 illustrates an embodiment according to one aspect of thedisclosure in which additional content is displayed on more than onedisplay screen associated with a user. In this embodiment, displaydevice 1402, camera 1404, and second display device 1422 are allcommunicatively coupled to processor 1516. As with the aboveembodiments, processor 1516 may be local to any device pictured, or maybe, e.g., a centralized server. Further, display device 1402, camera1404, and second display device 1422, may be communicatively coupled byany means well known and appreciated by those skilled in the art. Thisembodiment, using one of the various techniques as explained above,recognizes brand name object 1510 on cola can 1410, as represented bybroken lines 1508, and logo object 1506, as represented by broken lines1504. Processor 1516 processes these recognized objects as representedby processing cloud 1502 by, e.g., any of the aforementioned methods,and returns additional content 1512 and 1514 associated with the brandname object 1510 and/or logo object 1506. However, in this embodiment,processor 1516 returns additional content 1512 and 1514 to more than onedisplay device associated with a user. The additional content 1512 and1514 may be, e.g., the same on each display device (e.g., a commonadvertisement) or may differ on each display device. For example, in theembodiment depicted in FIG. 15, processor 1516 returns additionalcontent 1512 to display device 1402, and returns additional content 1514to second display device 1422. In this example, the content iscomplementary to, but different from, content that is being displayed onthe display device 1402. As another example, on display device 1402, thefirst additional content notifies user 1406 that they have just earnedABC Bucks. This may be, e.g., in connection with a loyalty program.Thus, when camera 1404 captures images containing objects associatedwith ABC Cola, ABC Cola may reward user 1406 by returning loyalty points(in this example referred to as “ABC Bucks”). Additional content 1514,displayed on second display device 1422, notifies user 1406 that ABCBucks have been added to a user account, and provides a link 1518 forthe user 1406 to access the account. Those skilled in the art willappreciate that additional content 1512 and additional content 1514 neednot be delivered and/or accessed at the same time. For example, in oneembodiment, the second additional content 1514 may be delivered via,e.g., email, and accessed by user 1406 at a later time. Alternatively,the second additional content 1514 may not be displayed until, e.g.,user 1406 accesses her account at a later point in time.

The various embodiments of the present disclosure as discussed hereinmay be employed within the embodiment depicted in FIG. 15. For example,the embodiment may instead capture data related to what is being viewedby user's 1406 eyes and determine what objects she is viewing. Forexample, returning to FIG. 14, user 1406 is generally viewing the leftside of display device 1402, as represented by broken lines 1420. A datacapture device, such as, e.g., camera 1404, may capture data related towhat is being viewed by user's 1406 eyes and determine, e.g., that sheis generally viewing the lower, left side of display device 1402. Thecontent 1520 being displayed at that time may include one or moreobjects. For example, content 1520 may include an advertising sign 1522and spectator 1524. Upon determining that user 1406 is generally viewingthe lower, left side of display device 1402, the embodiment maydetermine that this is associated with advertising sign 1522, whichdepicts an advertisement for ABC Cola. User 1406 may then be, e.g.,rewarded for spotting ABC Cola object within content 1520, andadditional content 1512 may be returned to display device 1402 andadditional content 1514 may be returned to second display device 1422 tonotify user 1406 of the reward. Alternatively, the embodiment of FIG. 15may process captured images associated with more than one user, aspresented in various above embodiments, and return additional content tomore than one display device accordingly.

In the above examples, additional content need not be returned to bothdisplay devices. For example, in some embodiments, additional contentmay only be returned to a second display device. Thus, in the exampledepicted in FIG. 15, additional content may only be returned to seconddisplay device 1422. That is, camera 1404 may capture an image, andprocessor 1514 may process a recognized object from the captured object(e.g., brand name object 1510 or logo object 1506), but, rather thanreturning both additional content 1512 and additional content 1514, theembodiment may only return, e.g., additional content 1514. In thisembodiment, user 1406 may be presented with additional content 1514immediately (e.g., by email, popup window, text message, etc.) or may bepresented with additional content 1514, e.g., next time she uses seconddisplay device 1422. Thus, in this embodiment, camera 1404 may capturean image containing objects surrounding user 1406 accessing content ondisplay device 1402 and/or capture the position of user's 1406 eyes anddetermine objects on display device 1402 the user is viewing, but,rather than return additional content to display device 1402, additionalcontent is returned to second display device 1422 (e.g., desktopcomputer, laptop computer, smartphone, tablet computer, and the like).User 1406 may then access this additional content either immediately or,e.g., the next time user 1406 uses second display device 1422.

FIGS. 16a-16d illustrate example embodiments of user interfaces whichmay be associated with the present disclosure. In FIG. 16a , display1600 contains profile creation menu 1602. Profile creation menu 1602 mayallow a user to create a profile according to one embodiment of thepresent disclosure. Profile creation menu 1602 contains radio buttonsallowing a user to indicate whether or not she would like to setup aprofile. If she would, profile creation menu 1602 may allow the user tostore, e.g., her name, email, and password for accessing her profile ata later date. Those skilled in the art will recognize many commonfeatures of a profile creation menu 1602 that may be incorporatedherein. For example, profile creation menu 1602 may further allow a userto upload an image, or avatar, to be associated with the profile.Profile creation menu 1602 may also allow a user to create a uniqueidentifier, or profile name, to be used at subsequent logins.

Turning to FIG. 16b , two additional user interfaces are depictedaccording to aspects of the disclosure. Specifically, captureauthorization menu 1604 and capture scheduler menu 1606 are displayed ondisplay device 1600. If a user has created and stored a profile, such asby using the profile creation menu of 1602, a user may enter herpreferences in capture authorization menu 1604 and capture schedulermenu 1606 only periodically, with processor 604 or the like storing herpreferences for reference each time she accesses content. Alternatively,the user may specify her preferences each time she access content, suchas when she powers on a display device 1600. Those skilled in the artwill appreciate that a number of ways prompting a user, receiving userinput, and storing or using user input may be employed without departingfrom the scope of this disclosure.

Capture authorization menu 1604 prompts a user for approval to captureimages by a camera associated with display device 1600. For example,some users may not want a camera to capture images of them and theirsurroundings when they are accessing content. Accordingly, a user couldsetup a profile and select “No” in response to “OK To Capture Images?”Alternatively, a user may not care if a camera captures images of themand their surroundings while accessing content, and accordingly a usercould select “Yes” in response to “OK To Capture Images?” In someembodiments, such a message may be displayed each time the processorwishes to use a captured image, and the user may be given the option ofdenying approval for a particular captured image after seeing what hasbeen captured.

Capture scheduler menu 1606 may allow a user to customize times when acamera associated with display device 1600 captures images. For example,if a user does not care if a camera captures images of her and hersurroundings while she accesses content, but the user does prefer not tohave images captured during a certain time, capture scheduler menu 1606allows the user to input preferable times for image capture. In oneembodiment, capture scheduler menu 1606 may further include, e.g., acalendar which allows a user to authorize image capture according tospecific days of the week. In an additional embodiment, a user may notcare if a camera captures an image of her surroundings, but prefers thecamera does not capture an image of her. In this embodiment, capturescheduler menu 1606 may further include an option which allows a user tospecify that images may only be captured when she leaves the room and/orfield of view of the camera.

Personalized message 1608 in FIG. 16c presents yet another embodiment ofa user interface according to some embodiments of the disclosure.Specifically, a content provider (in this example, ABC Cola) sends auser personalized message 1608 inviting the user to, e.g., create aprofile. For example, in the embodiment depicted, personalized message1608 presents an invitation from ABC Cola for the user to “Start EarningABC Bucks.” This may be sent in response to, e.g., processor 604determining that a recognized object in a captured image is associatedwith ABC Cola on a target list. Alternatively, this may be sent inresponse to, e.g., processor 604 determining that a user is viewingobjects on display device 1600 that is associated with ABC Cola.Further, in the embodiment illustrated in FIG. 16c , personalizedmessage 1608 contains a link 1612, which allows a user to access anothermenu (in this embodiment to, e.g., sign up for an ABC rewards account).Link 1612 may be any link as is well known to those in the art. Forexample, link 1612 may be a hyperlink displayed on a personal computer,tablet computer, or smartphone display screen. Accordingly, a user mayselect the link using, e.g., a mouse, keyboard, touch screen, or anyother input device, after which they will be directed to another screen.Alternatively, the link may appear on, e.g., a television screen and beselected by the user using, e.g., a remote control or other inputdevice. Those skilled in the art will also appreciate that links, suchas link 1612, can be associated with any of the embodiments as depictedthroughout the disclosure. For example, a link could be incorporatedwith, e.g., additional content 804 as depicted in FIG. 8, additionalcontent 1106 as depicted in FIG. 11, or additional content 1514 asdepicted in FIG. 15.

Returning to FIG. 16c , in one embodiment, when a user accesses link1612 they are redirected to a different screen or menu. For example,link 1612 may redirect a user to a menu where they can create a profile,such as profile creation menu 1202. Alternatively, the user may bedirected to a screen that contains, e.g., a coupon or other downloadableor otherwise accessible content.

Turning to FIG. 16d , an example feedback menu 1610 is presented.Feedback menu 1610 may include, e.g., a representation 1614 depicting arecognized image. Feedback menu 1610 may present the user withrepresentation 1614 of a recognized object, alerting her that the systemhas captured an image containing the recognized object. In thisembodiment, feedback menu identifies to the user that representation1614 is associated with the ABC Cola. Feedback menu 1610 may furtherprovide a user with options regarding the additional content associatedwith the object as depicted in representation 1614. For example,feedback menu 1610 may allow a user to opt out of receiving content froman entity associated with the object. Alternatively, feedback menu 1610may allow a user to opt in to future content from an entity associatedwith the object regardless of any subsequent matches. Those skilled inthe art will appreciate a myriad of other options that may be presentedto a user through feedback menu 1610 without departing from thedisclosure. Further, if a user has, e.g., a profile or other accountestablished, such as by, e.g., completing the profile creation menu1602, one embodiment of the disclosure may remember the user'spreferences, as entered through feedback menu 1610, when accessingfuture content. For example, if the user selects the option “NeverDisplay Content From ABC” when logged in to, e.g., a profile or useraccount, processor 604 may not return additional content in the futurewhenever the user is logged in even if a subsequent recognized objectmatches a target object on the target list associated with ABC Cola.

FIG. 17 depicts a flowchart for an exemplary method according to oneembodiment of the disclosure, which can be performed by one or morecomputing devices such as a gateway 111, display device 112, computingdevice 114, server 107, or any other desired computing device. Whendescribing the steps of the method below, the term “system” will be usedwhen referring to each component, either singularly or collectively,employed at each step of the method. Such usage is merely for ease indiscussing the method, and should not be interpreted as limiting theexact embodiment in which the following method may be performed.

At step 1702, a user is associated with a consumption device such asdisplay device. In some embodiments, a user may be associated with thedisplay device merely by turning on the display device in order toaccess content. In other embodiments, a user may, e.g., login to aprofile associated with her. That is, a user may have set up a profileusing, e.g., any of the user interfaces presented above and thus mayhave stored preferences regarding, e.g., when a camera may captureimages of her accessing content. In still another embodiment, a user maybe prompted at step 1702 to create a profile and store preferences. Inthis embodiment, when a user powers on a display device, she may bepresented with a user interface asking for, e.g., a user name andpassword in order to access preferences in a stored profile. If the userhas not yet setup a profile, she may be presented with an option tocreate a profile. At this time, the user may thus input informationabout herself and/or image capture preferences. For example, aspresented above in connection with FIGS. 16a -16 d, the user may specifyshe only wants images captured during certain times of the day, or,e.g., when she is accessing a certain type of content. If the userstores these preferences at step 1702, the system may then access suchpreferences at a later time as will be discussed in more detail below.Alternatively, if the user is not prompted to enter login information atstep 1702 and/or the user declines to login or setup a profile, thesystem may act in a default mode, wherein images are captured at alltimes or captured according to, e.g., a preloaded default image capturescheduler.

At step 1704, the system presents content to a user through a displaydevice. Such a display device may be any device, including, but notlimited to, a television, desktop computer, laptop computer, tabletcomputer, smartphone, and the like. Further, the content may containobjects as discussed more fully above. In such embodiments, the contentmay be provided with data (e.g., metadata, manifest file, script, etc.)indicating where in the content each object is located. At step 1706,the system associates a camera (or any image or data sensing orcapturing device) with the display device. As presented above, suchcamera may be integral to the display device, as is common in, e.g.,televisions, personal computers, tablet computers, smartphones, and thelike, or the camera may be external to the display device andoperatively coupled to the display device.

At step 1708, the system determines whether or not the camera isauthorized to capture an image. As presented above, when a user isassociated with the display device at 1702, she may have entereduser-specific preferences regarding when the system is authorized tocapture images, and such preferences are referenced at step 1708 todetermine if the system is authorized to capture an image at this time.Alternatively, the user may have merely, e.g., powered on the displaydevice and not entered user-specific preferences. Thus, the system maydetermine it is authorized to always capture images and/or check adefault image capture scheduler to determine if it is currentlyauthorized to capture images. For example, in one embodiment, a user mayhave not authorized images to be captured of her and her surroundingswhile accessing content. In another embodiment, a user (and/or a defaultscheduler) might have specified certain times when the camera isauthorized to capture images of her and her surroundings while accessingcontent, and the system determines if the current time is within anauthorized window. In still another embodiment, a user (and/or a defaultscheduler) may have specified that the camera is only authorized tocapture images once the user has left the room, and the systemdetermines whether or not the user has left the room before capturing animage. In still another embodiment, the user (and/or a defaultscheduler) may have specified certain content during which the system isauthorized to capture an image, and the system will determine what typeof content is being accessed on the display device and whether or not itis authorized to capture an image during the displayed content. Evenwhen a user profile permits it, the capturing of an image need not occurcontinuously. Instead, the device may capture images only atpredetermined time intervals (e.g., once every 10 minutes), or inresponse to certain events (e.g., initial turning on of display device,changing channels, etc.). If the system determines it is not authorizedto capture an image, or that an image is not to be captured, the methodproceeds to step 1710. If the system determines it is time to capture animage, the method proceeds to step 1712.

At step 1710, the system determines whether or not it should continue tocapture images. For example, in one embodiment the display device may bepowered off by the user, and in such an event the system will determineit should no longer capture images and the method thus ends. In anotherembodiment, the user may disable the camera and/or capture function(through, e.g., a user interface) and the method thus ends. In anyevent, when the system determines it should no longer capture images,the method ends. However, if the system determines it should continue tocapture images, the method returns to step 1708. For example, the systemat step 1708 may have determined it is not currently authorized tocapture images because, e.g., the current time is not within auser-specified authorized window, or the content the user is accessingis not content during which the system is authorized to capture animage, or the user is currently in the room and the user has notauthorized image capture while the user is in the room. In suchexamples, the system will determine at step 1708 that it is notauthorized to currently capture an image, but will determine at step1710 that it should continue to attempt to capture images. Thus, themethod continually cycles through steps 1708 and 1710 until the systemis ultimately authorized to capture an image or the system determines itshould no longer attempt to capture images and thus the method ends.

Once the system determines it may capture an image at 1708, the methodproceeds to step 1712. In step 1712, the system captures an image orother data using the camera or other image or data capture device. Inone embodiment, such an image may be, e.g., of a user and hersurroundings while the user is accessing content. In other embodiments,the image may be, e.g., of a room after the user has left. In stillother embodiments, data related to what is being viewed by the user'seyes may be captured. In some embodiments, a user may be presented withthe image for approval at step 1714. In one embodiment, the user mayreceive a prompt on the display device informing her that an image hasbeen captured, and the user may be asked for permission to furtherprocess the image. In another embodiment, the user may be presented witha representation on the screen of the captured image and may be askedfor permission to further process the image. Thus, at step 1714 the usermay be able to immediately determine if she wants to further process theimage. If, for any reason, the user does not want the system to processthe image, the user at step 1714 can decline permission to proceed. Insuch an event, the captured image may, e.g., be immediately deleted, andthe method would return to step 1710 and proceed through each step aspresented herein. However, if the user approves of the system furtherprocessing the image, she may give her permission at step 1714 for thesystem to use the image, and the method proceeds to step 1716.

In step 1716, the system processes the image or data to determine and/orrecognize any objects contained within the captured image and/or thegaze of the user's eyes (e.g., what portion of the content the user isviewing). This may be accomplished by, e.g., any method of objectrecognition which is well known in the art and/or as described above. Ifthe system does not recognize an object and/or what portion of thecontent the user is viewing by, e.g., any of the aforementionedtechniques, the method proceeds to step 1710 and the system determines,as explained above, if it should capture further images or data.However, if the system recognizes one or more objects or the portion ofthe content the user is viewing at step 1716, the method proceeds tostep 1718.

At step 1718, the system determines if it is authorized to use therecognized object. For example, as presented in the various embodimentsabove, a user may have a profile associated with her. In such a profile,the user may have, e.g., specified certain objects that she does notwant to be used to return additional content. Returning to FIG. 16d ,the user may have specified, in response to, e.g., additional content1610, that she does not wish to receive additional content associatedwith objects resembling ABC Cola. In such an embodiment, the system atstep 1718 will not use a recognized object if associated with ABC Cola,and thus the method returns to step 1710 and proceeds through each stepas described herein. Alternatively, the system may prompt the user at1718 and ask her whether or not the system may use the object to searchfor and/or return additional content. For example, a representation ofthe recognized object may be presented to the user through the displayscreen, and the user may indicate whether or not the system isauthorized to process the recognized object and search for and/or returnadditional content accordingly.

If the system is authorized to use the recognized object at step 1718,the method proceeds to step 1719. At step 1719, the system may checkdata associated with the content, such as a script. For example, thecontent provided may have data associated with it (e.g., a script,metadata, manifest file, etc.) which provides, e.g., what objects arelocated within the content and/or the location of the objects. At step1719, the system may thus access the data associated with the content(e.g., by referencing a script, metadata, manifest file, or by searchinga network, etc.) to see if objects are associated with the content. Atstep 1721, the system determines, from accessing the data at 1719,whether or not objects are included in the content. If objects are notincluded, the method proceeds to step 1720. However, if objects areincluded, the method proceeds to step 1723, where the system may storeor transmit information associated with the objects. For example, at1723 the system may store information regarding if and how long the userviewed the object to be later used to, e.g., charge an advertiser asdiscussed above.

After the system stores and/or transmits information relating to theuser's interaction with one or more objects, the method proceeds to step1720. In step 1720, the system compares the recognized object and/ordetermined objects the user interacted with to target objects on atarget list (this step, and the processing of objects/objects herein,may be repeated for each identified object). The target list may beeither local to, e.g., the camera and/or display device, or the targetlist may be located remotely from the camera and/or display device. Forexample, in some embodiments the target list may be stored in memoryoperatively coupled to a processor that is integral to, e.g., the cameraand/or the display device. In another embodiment, the system maytransmit the recognized object to, e.g., a centralized server. In anyevent, the objects as recognized in step 1716 are compared to a targetlist either locally or remotely. Optionally, data related to what isbeing viewed by a user's eyes may be used at step 1720 to determine,e.g., objects on the display screen the user is viewing using any methodas discussed more fully above. An object the user is viewing on thescreen may then be compared to the target list at step 1720. At step1722, the system determines whether or not the recognized object matchesa target object in the target list. If the recognized object matches,beyond a minimum threshold similarity match, a target object in thetarget list, the method proceeds to step 1732. If, however, therecognized object does not match (or, alternatively, does not fullymatch as will be discussed below) a target object in the target list,the method proceeds to step 1724.

At step 1724, the system receives information about the recognizedobject in an attempt to identify the object. For example, in oneembodiment the system may return an imperfect match at step 1722. Thatis, the image may not be a close enough match to be deemed a strongmatch, but it may be close enough to be a weak match. For example,returning to FIG. 5, the system may match brand name object 410 to atarget object associated with ABC Cola because the system has aprestored image of the can that matches well, but may not immediatelymatch logo object 404 on the hat to ABC Cola because the system does nothave a stored image of the hat with the logo (e.g., perhaps the systemonly has a stored image of the logo itself, without the hat). In oneembodiment, the system may determine that the logo object 404, althoughnot identically matching with its prestored images, is at least apartial match based on the ABC logo. The system may then return to user306 a prompt, containing, e.g., a representation of logo object 404, andask the user whether the recognized object is associated with theimperfectly matched target object, here a target object associated withABC Cola. The user may then be able to either confirm or deny the objectis associated with the imperfectly matched target object.

In other embodiments, such as when, e.g., the system has not returned animperfect match, the user may be prompted to input information regardingthe unmatched recognized object at step 1724. For example, returning toFIG. 6c , the user may be presented with, e.g., representation 618 of anunrecognized object. In such an embodiment, the user may be furtherprompted to enter information about the object. For example, the user,when presented with a representation of photograph object 408, may inputinformation such as “mountains” or “Denver, Colo.” Alternatively, theuser may enter, e.g., a URL associated with the object or any otherinformation to assist the system in matching the object. The system maythen use this inputted information to attempt to match the object. Forexample, the system may compare the inputted keywords to keywords ofstored target objects on the target list. If the keywords lead to animperfect match (i.e., the system guesses a target object may beassociated with the recognized object) the user may then be furtherprompted to confirm the imperfect match as presented above. In otherembodiments, the system may send the representation to a remotecomputing device comprising a broader database in an attempt to matchthe target object. In still another embodiment, the system may use,e.g., web queries or internet searches to ultimately match therecognized object to a target object.

If the recognized object is ultimately identified, the system may then,optionally, store the object and/or associated information in the targetlist at step 1730. In this manner, the object may be more easily matchedin the future and/or may have additional content associated with it forfuture matches. In order to facilitate such broadening of the targetlist in each embodiment, the user may be, e.g., rewarded for helpingidentify the recognized object. For example, as presented above, a usermay have set up a profile associated with a loyalty program for certainentities. Upon helping identify new objects associated with that entity(by, e.g., confirming a partial match or entering information for anunmatched object), the user may receive loyalty points or otherpromotional offers. In this way, users may be incentivized to helpimprove a target list, and thus the target list may be ever-evolving.The system may also catalogue the locations, devices, addresses (e.g.,IP addresses) of the places where matches were found. The aggregatedinformation may be stored and presented to the user if desired. Theinformation can identify, for example, the percentage of a user'sneighbors who have the same soda can in their house. After the object isstored in the target list, the method proceeds to step 1732 which willbe discussed more fully below.

If the recognized object is ultimately not identified, the system mayreturn an unidentifiable message at step 1728. In one embodiment, theunidentifiable message may be returned to a user. In such an embodiment,the user may be presented with a representation of the object and amessage indicating the object was not successfully identified using anyof the techniques as discussed above. Alternatively or additionally, anunidentifiable message may be sent to, e.g., a centralized server. Insuch an embodiment, a representation of the unidentified object and anyinformation inputted by the user may be returned to a server. Thisinformation later may be used to ultimately identify the object andupdate the target list for future applications. In any event, if theobject ultimately remains unidentified, the method, after the systemreturns an unidentified object message at step 1728, returns to step1710, and proceeds through each step as described herein.

Returning now to step 1722, if the system does match the object to atarget object on the target list, the method proceeds to step 1732.Further, if the system ultimately identifies the object at step 1726(through either, e.g., the assistance of a user or a successful webquery), the method ultimately proceeds to step 1732 after, optionally,storing the object in the target list at step 1730 as described above.At step 1732, the system determines whether there is additional contentassociated with the target object on the target list. Some objects maybe successfully matched to target objects at 1722, or alternativelysuccessfully identified at step 1726, however, no additional contentwill be associated with the target object on the target list. Forexample, in the various examples depicted above, in some embodimentsthere may not be any content associated in the target list with, e.g.,graphic object 406. That is, the method may recognize graphic object406, and the method may even match graphic object 406 to an object onthe target list (namely, an object associated with State University).However, State University (or any other entity for that matter) may nothave yet associated additional content with the target object. In suchan embodiment, therefore, the system will not have additional content toreturn to a display device. The method proceeds to step 1738, where thesystem processes the target. At step 1738, the system may, e.g., updatea count of how many times the target object has been matched. That is,the system may track the popularity of certain target objects, andentities may later access such statistics and ultimately decide whetheror not to associate additional content with the target object. Themethod may also return a message to entities associated with the targetobject alerting them that the target object has been matched. In thisway, entities may continually receive updates regarding how oftencertain objects are recognized in captured images, and may thus, e.g.,base future decisions regarding which objects to associate additionalcontent with accordingly.

However, if additional content is associated with the target object inthe target list, the method proceeds to step 1734. In step 1734, thesystem delivers additional content. That is, as described in detailabove, the system will return the additional content associated with thetarget object to the display device where it will be displayed to theuser, or send an email to the user, or otherwise deliver the additionalcontent to the user. This may be accomplished using any of the variouslydescribed methods throughout the present disclosure. For example, thesystem may display an advertisement on the display screen associatedwith the recognized object, or the system may return a personalizedmessage on the display screen and/or a second display screen, or thesystem may return additional content based on the number of recognizedobjects associated with an entity in the user's room compared to, e.g.,the number of recognized objects in other's rooms within the user'sneighborhood. In some embodiments, the system use physical proximity ofdevices to determine the best route for delivering the additionalcontent. For example, the system may determine that the user has asecond screen device 1422 in the room with the display, and rather thanoverlaying the additional content on the main display, the system canchoose to send the additional content to the second screen device 1422instead. Other homes lacking the second screen device 1422 might stillhave the additional content appear on their primary displays. In otherembodiments, the system may graphically overlay additional content on ageneric or other product placement, and/or scramble other objectslocated in the content. For example, in order to offer an advertiserexclusivity, a system may scramble or otherwise remove any other brandsor logos appearing in the content that is not complimentary to theadditional content. Or the system may graphically overlay additionalcontent on generic products placed within the delivered content. Any ofthe aforementioned methods of returning additional content, as describedin detail above, may be used at step 1734.

At step 1736, a user's profile may be updated. For example, informationregarding what objects were matched and/or what additional content wasdelivered may be stored in a user's profile. Accordingly, a system mayreference such information when returning additional content in thefuture. For example, an entity who has associated content with a targetobject on the target list may specify, e.g., a limit to how many timesadditional content may be returned in a given time period. Thus, whenreturning additional content in the future, the system may firstdetermine whether or not a maximum amount of times has been reached.Further, as discussed in relation to the, e.g., loyalty programs above,loyalty points may be added and/or promotional offers returned to auser's profile in connection with the returned additional content.Further, information relating to objects recognized and content returnedmay be stored in a user's profile. In this embodiment, a user could thusperiodically review objects that have been recognized and additionalcontent that has been returned.

Once the user profile is updated at step 1736, the method proceeds tothe step 1710 wherein the system determines whether or not it shouldcontinue to capture images. If yes, the method proceeds through eachstep as indicated above. If no, the method ends.

Although example embodiments are described above, the various featuresand steps may be combined, divided, omitted, and/or augmented in anydesired manner, depending on the specific outcome and/or application.Various alterations, modifications, and improvements will readily occurto those skilled in art. Such alterations, modifications, andimprovements as are made obvious by this disclosure are intended to bepart of this description though not expressly stated herein, and areintended to be within the spirit and scope of the disclosure.Accordingly, the foregoing description is by way of example only, andnot limiting. This patent is limited only as defined in the followingclaims and equivalents thereto.

1. A system comprising: a first image capture device configured to,during output of content by a first computing device, capture a firstimage of a first field of view; a second image capture device configuredto, during output of the content by a second computing device, capture asecond image of a second field of view different from the first field ofview; and a third computing device, separate from the first computingdevice and the second computing device, configured to: access, inmemory, object recognition data associated with recognition of aplurality of candidate objects; determine, based on the objectrecognition data, a first quantity of occurrences of an object in thefirst image and a second quantity of occurrences of the object in thesecond image; and output, to one of the first computing device or thesecond computing device based on whether the first quantity ofoccurrences or the second quantity of occurrences is greater, additionalcontent comprising a message about the object.
 2. The system of claim 1,wherein the additional content comprises an advertisement for theobject, and wherein the third computing device outputs the additionalcontent at a commercial break of the content.
 3. The system of claim 1,wherein the first computing device is configured to output a messageasking a user to input information about an unidentified objectappearing in the first image, and wherein the third computing device isconfigured to determine, based on the information, that the unidentifiedobject is the object.
 4. The system of claim 1, wherein the firstcomputing device is configured to output a message requesting anindication of a time range during which the first image capture deviceis authorized to capture an image.
 5. The system of claim 1, wherein thethird computing device is configured to determine, based on contactinformation, to compare the first quantity of occurrences with thesecond quantity of occurrences.
 6. The system of claim 1, wherein thethird computing device is configured to: determine, based on userinformation associated with the first computing device, a contact listassociated with a social media site; and determine, based on the contactlist, to compare the first quantity of occurrences with the secondquantity of occurrences.
 7. The system of claim 1, wherein the thirdcomputing device is configured to determine, based on a geographiclocation of the first computing device, to compare the first quantity ofoccurrences with the second quantity of occurrences.
 8. The system ofclaim 1, wherein the first image capture device is configured todetermine a first duration that a first user views a first portion ofthe content, wherein the second image capture device is configured todetermine a second duration that a second user views a second portion ofthe content, and wherein the third computing device is configured tooutput, to one of the first computing device or the second computingdevice based on whether the first duration or the second duration isgreater, second additional content.
 9. The system of claim 1, whereinthe first computing device is configured to send the additional contentto a user device.
 10. The system of claim 1, wherein the first computingdevice is configured to output the additional content for display overthe content.
 11. A system comprising: a network interface configured tocommunicate with a plurality of first computing devices and send contentto the plurality of first computing devices; a second computing device,in communication with the network interface, configured to: receive, viathe network interface, a first image, captured by a first image capturedevice and a second image captured by a second image capture device,wherein the first image and the second image are different and arecaptured during output of the content by the plurality of firstcomputing devices; determine, based on object recognition dataassociated with recognition of a plurality of candidate objects, a firstquantity of occurrences of an object in the first image and a secondquantity of occurrences of the object in the second image; and output,to one of the plurality of first computing devices based on whether thefirst quantity of occurrences or the second quantity of occurrences isgreater, additional content comprising a message about the object. 12.The system of claim 11, wherein the second computing device isconfigured to: receive, via the network interface, a first duration thata first user of one of the plurality of first computing devices views afirst portion of the content; receive, via the network interface, asecond duration that a second user of another one of the plurality offirst computing devices views a second portion of the content; andoutput, to one of the one of the plurality of first computing devices orthe another one of the plurality of first computing devices based onwhether the first duration or the second duration is greater, secondadditional content.
 13. The system of claim 11, wherein the one of theplurality of first computing devices is configured to send theadditional content to a user device.
 14. The system of claim 11, whereinthe second computing device is configured to: generate a message toprompt a user to input information about a candidate object in the firstimage; receive the information; and update, based on the information,the object recognition data to facilitate recognition of the candidateobject.
 15. The system of claim 11, wherein the second computing deviceis configured to control the first image capture device to capture thefirst image.
 16. A system comprising: a network interface configured tocommunicate with one or more servers and receive content that istransmitted to a plurality of computing devices; a first image capturedevice configured to capture images; and a first computing deviceconfigured to: control the first image capture device to, during outputof the content, capture a first image of a first field of view;determine, based on object recognition data associated with recognitionof a plurality of candidate objects, a first quantity of occurrences ofan object in the first image; send, via the network interface and to theone or more servers, the first quantity of occurrences, wherein the oneor more servers are configured to: compare the first quantity ofoccurrences with a second quantity of occurrences of the object in asecond image captured by a second image capture device associated with asecond computing device that is different from the first computingdevice; and output, to one of the first computing device or the secondcomputing device based on whether the first quantity of occurrences orthe second quantity of occurrences is greater, additional contentcomprising a message about the object; and output, to a display, theadditional content.
 17. The system of claim 16, wherein the firstcomputing device is configured to: prior to controlling the first imagecapture device to capture the first image, determine whether the firstimage capture device is authorized to capture the first image.
 18. Thesystem of claim 16, wherein the first computing device is configured to:prior to controlling the first image capture device to capture the firstimage, prompt a user to identify a time range during which the firstimage capture device is authorized to capture the images; and at a startof the time range, activate the first image capture device.
 19. Thesystem of claim 16, wherein the first computing device is configured to:prior to controlling the first image capture device to capture the firstimage, determine that a user has left the first field of view.
 20. Thesystem of claim 16, wherein the first image capture device comprises acamera that is integral to the first computing device.